Automatic creation of a scalable relevance ordered representation of an image collection

ABSTRACT

In a method of automatically creating a scalable relevance ordered representation of an image collection, the images in the image collection are classified into a plurality of clusters based upon a feature of the images. In addition, respective relevance levels of the images contained in each of the plurality of clusters are determined and the images in each of the plurality of clusters are ordered according to the relevance levels. Moreover, the images from the ordered plurality of clusters are arranged according to a predefined arrangement process to create the scalable relevance ordered representation of the image collection.

RELATED APPLICATIONS

The present application shares some common subject matter with copendingand commonly assigned U.S. patent application Ser. No. 11/127,079,titled “METHOD AND SYSTEM FOR AUTOMATICALLY SELECTING IMAGES FROM AMONGMULTIPLE IMAGES”, filed on May 12, 2005, and copending U.S. ProvisionalPatent Application Ser. No. 61/012,099, titled “PROCESSING PIXEL VALUESOF A COLOR IMAGE”, filed on Dec. 7, 2007, the disclosures of which arehereby incorporated by reference in their entireties.

BACKGROUND

With the advent and proliferation of digital cameras and videorecorders, along with increased data storage capabilities atincreasingly lower costs, it has become common for users to collect everincreasing numbers of images in a collection. For instance, it is notuncommon for users to take hundreds of digital images during a singleevent, such as, a wedding, a vacation, and a party. When a user wishesto create a photo album, photobook, or a slideshow containing some ofthe images, the user typically arranges the photographs in chronologicalorder, based on scene content, or the person who captured thephotographs. However, due to the relatively large number of images,users often spend a great deal of time in sorting through the imagecollection to determine which of the images to include.

Conventional systems for automatic image classification have beenapplied to multiple images stored in a database. The classification hasbeen used to index images so that the images may be categorized,browsed, and retrieved. In addition, images have been stored in thedatabase with descriptive information regarding the image file, such as,the file creation date, file name, and file extension. Techniques usedfor image classification are, for the most part, similar toclassification techniques applied to any form of digital information.

An exemplary image classification technique provides for navigationthrough a collection of images to facilitate image retrieval. Theappearance of an image is summarized by distribution of color or texturefeatures, and a metric is defined between any two such distributions. Ameasure of perceptual dissimilarity is provided to assist in imageretrieval. Two or three-dimensional Euclidean space has been used toevaluate differences in distances between images to highlight imagedissimilarities. The results may be used to assist in a database queryfor locating a particular image.

Although conventional image classification techniques provide users withthe ability to more easily navigate through images, these techniquesstill require that users expend a great deal of manual effort inselecting desired images, such as, images representative of particularevents, because the users are still required to manually search throughthe images.

An improved approach to selecting good representative images from animage collection for inclusion in a photo album, photobook, a slideshow,etc., that requires relatively less user input would therefore bebeneficial.

BRIEF DESCRIPTION OF THE DRAWINGS

Features of the present invention will become apparent to those skilledin the art from the following description with reference to the figures,in which:

FIG. 1 depicts a simplified block diagram of a computer-implementedsystem for creating a scalable relevance ordered representation of animage collection, according to an embodiment of the invention;

FIG. 2 depicts a flow diagram of a method of creating a scalablerelevance ordered representation of an image collection, according to anembodiment of the invention;

FIG. 3A depicts a graphical illustration of the steps contained in theflow diagram depicted in FIG. 2, according to an embodiment of theinvention;

FIG. 3B shows a graphical relevance representation of a step containedin the flow diagram depicted in FIG. 2, according to an embodiment ofthe invention; and

FIG. 4 shows a diagram of a sample hierarchical scalable imagecollection representation 310 and scalable relevance orderedrepresentations, according to an embodiment of the invention.

DETAILED DESCRIPTION

For simplicity and illustrative purposes, the present invention isdescribed by referring mainly to an exemplary embodiment thereof. In thefollowing description, numerous specific details are set forth in orderto provide a thorough understanding of the present invention. It will beapparent however, to one of ordinary skill in the art, that the presentinvention may be practiced without limitation to these specific details.In other instances, well known methods and structures have not beendescribed in detail so as not to unnecessarily obscure the presentinvention.

Described herein are methods and systems for automatically creating ascalable representation of an image collection. The scalablerepresentation is composed of an arrangement of images selected from animage collection that provides a desired result. For instance, thescalable representation is composed of the most relevant images in theimage collection that provide good coverage of particular events. Themost relevant images may comprise those images containing particularpeople, places, events, etc. In addition, the most relevant images maycomprise the most appealing images. As another example, the scalablerepresentation is composed of images arranged and ordered to tell astory based upon the order in which various events occurred. As afurther example, the scalable representation if composed of imagesarranged and ordered to provide coverage of images containing particularpeople, objects, places, etc.

Through implementation of the methods and systems disclosed herein, ascalable ordered representation of an image collection may automaticallybe created, such that, the scalable representation may easily be scaledto incorporate a desired number of images into at least one of adocument, a photobook, a photo album, a slide show, a calendar, etc.,while providing at least a desired level of coverage. The representationis easily scalable because the images are ordered in terms of relevanceand thus, the least relevant images may be easily removed to enable onlythe most relevant images to remain in the representation as desired.

With reference first to FIG. 1, there is shown a simplified blockdiagram of a computer-implemented system 100 for creating a scalablerelevance ordered representation of an image collection, according to anexample. In one regard, the various methods and systems disclosed hereinmay be implemented in the computer-implemented system 100 depicted inFIG. 1 as discussed in greater detail herein below. It should beunderstood that the system 100 may include additional components andthat some of the components described herein may be removed and/ormodified without departing from a scope of the system 100.

As shown in FIG. 1, the system 100 includes a computing apparatus 110,an input source 120, and an output 130. The computing apparatus 110includes a processor 112 and a memory 114, which stores an imagecollection 116. The processor 112 is configured to control variousoperations performed in the computing apparatus 110. One of theoperations includes the creation of a scalable relevance orderedrepresentation 118 of the image collection 116, which may also be storedin the memory 114. Alternatively, however, the image collection 116 maybe stored in a separate data storage device, such as, an external datastorage device, etc. In any regard, the image collection 116 maycomprise all of the image files contained in a single file folder, asubset of the all of the image files contained in the single filefolder, the images contained in a plurality of file folders, etc.

As referenced herein, an “image” is any image or electronic data filethat is stored in electronic form in any type of electronic storagemedium or computer memory. An image can be any digital image capturedfrom any type of digital capturing device, such as, without limitation,digital video cameras, digital still cameras, video capture cards, orother image capturing device. An image may alternately be an analogimage captured from, for example, any camera, video camera, or analogimage capture source or device, that are scanned or otherwise digitizedand stored as a digital image. An image may also be an electronicdocument, such as, for example, a multimedia document that containsimages, video, sound, etc. Those skilled in the art will recognized thatthe image can be any type of electronic file that a user may access froma collection of electronic data file.

Generally, speaking, the scalable relevance ordered representation 118of the image collection 116 comprises an arrangement of the images inthe image collection 116, such that, when the representation 118 isscaled to include fewer than all of the images, for instance, byremoving images identified as being relatively less relevant, theremaining images provide a relatively good coverage of all of theimportant events (faces, animals, places, objects, etc.) in the imagecollection. Various manners in which the processor 112 may create thescalable relevance ordered representation 118 are described in greaterdetail herein below.

According to an example, the processor 112 comprises a microprocessorcircuit programmed to create the scalable relevance orderedrepresentation 118. According to another example, code for creating thescalable relevance ordered representation 118 is stored as software onthe memory 114, which the processor 112 is configured to implement orexecute.

In any regard, the memory 114 comprises any reasonably suitable devicecapable of storage of information or any combination of devices capableof storage of information, such as, a semiconductor device, a magneticdisk memory device, nonvolatile memory devices, such as, an EEPROM orCDROM, etc. The memory 114 may also comprise a fixed or removable datastorage device. In addition to storing the image collection 116 and thescalable representation 118 of the image collection 116, the memory 114may also store one or more program instructions or code, which theprocessor 112 may execute in performing various operations of thecomputing apparatus 110.

The scalable relevance ordered representation 118 may comprise copies ofthe images contained in the image collection 116. Alternatively, therepresentation 118 may comprise indications of the images contained inthe image collection 116. In addition, the representation 118 need notbe stored as part of the memory 114, but may be stored in a separatelocation, and may include any information which will allows theprocessor 112 to retrieve, for sequential display, the selected images.For example, the selected images (or indications thereof), or portionsthereof, associated with the representation 118 may be stored within alocal memory of the processor 112.

The input source 120 may comprise a user interface, such as, a keyboard,mouse, touchscreen display, another computing apparatus, etc., that auser may use in inputting data into the computing apparatus 110. Forinstance, a user may employ the input source 120 to input informationpertaining to the number of images, the percentage of images from theimage collection, the objects contained in the images, etc., desired inthe representation 118 of the image collection 116. The user may alsoemploy the input source 120 to tag selected ones of the images as beinghighly relevant, identify desired parameters in the representation 118.

The output 130 may comprise any reasonably suitable apparatus to whichthe scalable representation 118 of the image collection 116 may beoutputted. The output 130 may thus comprise, for instance, a display, aprinter, another computing apparatus, a data storage device, a serverconnected to the computing apparatus 110 via the Internet, etc. By wayof example, the scalable representation 118 may be employed tosubstantially automatically create a photo-book, a slide-show, acalendar, a photo-album, etc., containing selected images from the imagecollection 116, which may be stored, displayed, and/or printed by theoutput 130.

Although not shown, the computing apparatus 110 may include additionalcomponents, such as, a communication bus and a secondary memory. Thecomputing apparatus 110 may also be interfaced with user input andoutput devices, such as, a keyboard, a mouse, and a display. Inaddition, the processor 112 may communicate over a network, forinstance, the Internet, LAN, etc., through a network adaptor.

An example of a method in which the computing apparatus 110 may beemployed to create a scalable relevance ordered representation 118 of animage collection 116 will now be described with respect to the followingflow diagram of the method 200 depicted in FIG. 2. It should be apparentto those of ordinary skill in the art that other steps may be added orexisting steps may be removed, modified or rearranged without departingfrom the scope of the method 200.

The description of the method 200 is made with reference to thecomputing apparatus 110 illustrated in FIG. 1, and thus makes referenceto the elements cited therein. It should, however, be understood thatthe method 200 is not limited to the elements set forth in the system100. Instead, it should be understood that the method 200 may bepracticed by a computing apparatus having a different configuration thanthat set forth in FIG. 1.

Some or all of the operations set forth in the method 200 may becontained as a utility, program, or subprogram, in any desired computeraccessible medium. In addition, the method 200 may be embodied bycomputer programs, which may exist in a variety of forms both active andinactive. For example, they may exist as software program(s) comprisedof program instructions in source code, object code, executable code orother formats. Any of the above may be embodied on a computer readablemedium, which include storage devices and signals, in compressed oruncompressed form.

Exemplary computer readable storage devices include conventionalcomputer system RAM, ROM, EPROM, EEPROM, and magnetic or optical disksor tapes. Exemplary computer readable signals, whether modulated using acarrier or not, are signals that a computer system hosting or runningthe computer program can be configured to access, including signalsdownloaded through the Internet or other networks. Concrete examples ofthe foregoing include distribution of the programs on a CD ROM or viaInternet download. In a sense, the Internet itself, as an abstractentity, is a computer readable medium. The same is true of computernetworks in general. It is therefore to be understood that anyelectronic device capable of executing the above-described functions mayperform those functions enumerated above.

The processor 112 may implement or execute the method 200 to create ascalable relevance ordered representation 118 of the image collection116. As will become clearer from the discussion below, therepresentation 118 is considered to be scalable because the processor112 is configured to order the images according to a relevance measure.As such, when the representation 118 is scaled down, there is lessimpact on the overall representation 118 because less relevant imagesare removed from the representation 118 first.

The method 200 will be described with reference to FIGS. 3A and 3B,which respectively depict a graphical illustration 300 of the stepscontained in the method 200 and a graphical relevance representation 312of step 206. Specific reference to the elements depicted in FIGS. 3A and3B are made for purposes of illustration and not of limitation. As such,the method 200 should not be construed as being limited to the elementsdepicted in FIGS. 3A and 3B.

At step 202, the processor 112 classifies the images contained in theimage collection 116 into a plurality of clusters based upon at leastone feature of the images. In one example, each of the plurality ofclusters comprises a different time period, for instance, based upon atemporal characteristic. In another example, each of the plurality ofclusters comprises a different actor or object. In yet a furtherexample, each of the plurality of clusters comprises a differentgeographic location, either as depicted in the image or based uponidentification data, such as, a global positioning system coordinateassociated with the different geographic locations. According to afurther example, the processor 112 may classify the images into clustersof more than one type, such as, both time clusters and actor clusters.It should be understood that the clusters may be divided based upon anyreasonably suitable characteristic(s) of either or both of the subjectmatter contained in the images and the identification data of theimages.

In instances where the feature of the images comprises time, theprocessor 112 may obtain the time information from an automaticallycreated time stamp associated with each image or from user-inputted timeinformation. In instances where the feature of the images comprisesactors, the processor 112 may execute or implement a face detectionalgorithm designed to identify faces in images and to also distinguishbetween the detected faces. An example of a suitable method fordetecting and distinguishing between faces is described in Xiao, J. andZhang, T., “Face Bubble: Photo Browsing with Faces”, Proceedings ofAdvanced Visual Interfaces 2008, Napoli, Italy, May 28-30, 2008, thedisclosure of which is hereby incorporated by reference in its entirety.Other types of objects in the images may be identified through similarmethods.

As shown in FIG. 3A, the processor 112 may classify the images of theimage collection 116 as a hierarchical image collection representation302 having different hierarchies of clusters 304, 306 a-306 c. Eachlevel of the hierarchy may include all of the same images, but dividedinto different numbers of clusters 304, 306 a-306 c. More particularly,for instance, the processor 112 may classify the images into a lessernumber of clusters 304 at a higher level of the hierarchy and into agreater number of clusters 306 a-306 c at a lower level of thehierarchy.

According to an example, the processor 112 arranges the clustershierarchically according to, for instance, time intervals in which theimages contained in the clusters were obtained. In this example, the tophierarchy is devised to accommodate the smallest sized representations118 in order to enable good coverage of images over a desired timeperiod, as discussed herein below.

The clusters may be divided into any suitable time periods, such as,years, months, days, time periods within days, etc. For instance, withreference back to FIG. 3A, the higher level cluster 304 may be one monthand the lower level clusters 306 a-306 c may be days or weeks in themonth. A suitable example of a manner in which the clusters may bedivided based upon temporal characteristics is discussed in the Ser. No.11/127,079 application for patent.

At step 204, in each of the plurality of clusters, the processor 112determines respective relevance measures of the images. The relevancemeasures of the images may comprise user-configurable relevance orderingbased on one or more features such as, an image appeal metric, faceclustering, smile detection, user-favorite tagging, and substantialsimilarities. The types of relevance measures applied and/or weightingapplied to various ones of the relevance measures may beuser-selectable. Thus, for instance, a user may indicate that imagescaptured during a particular time frame or at a particular location areto be given greater relevance over other images. In addition, oralternatively, a user may indicate that images containing particularfaces or objects are to be given greater relevance over other images.

Thus, by way of example, the processor 112 may process all of the imagesto determine one or more characteristics of the images and to determinerespective relevance measures of the images. The one or morecharacteristics may include, for instance, whether the images containfaces, whether in those images containing faces, the actors are smiling,whether the images contain particular objects, etc. In cases where theprocessor 112 is configured to detect whether images contain smilingfaces, the processor 112 may employ a facial expression recognitionalgorithm. An example of a suitable facial expression recognitionalgorithm is described in Chen, X., and Huang, T. “Facial ExpressionRecognition: a Clustering-Based Approach”, Pattern Recognition Letters,v.24, n. 9-10, p. 1295-1302, Jun. 1, 2003, the disclosure of which ishereby incorporated by reference in its entirety. The processor 112 mayalso employ a suitable object detection algorithm to detect particularobjects in the images.

The processor 112 may further implement an image appeal metric todetermine image appeal values of each of the images contained in theimage collection 116. In addition, the processor 112 is configured touse the image appeal values in determining respective relevance levels.For instance, the processor 112 is configured to assign higher relevancelevels to images having higher image appeal values.

Generally speaking, “image appeal” may be defined as the interest thatan image generates when viewed by human observers, incorporatingsubjective factors on top of the traditional objective quality measures.According to an example, the processor 112 automatically determinesimage appeal values for the images through implementation of an imageappeal metric as described herein below.

In this example, for each of the images, the processor 112 assigns arepresentative sharpness (S) value to each region of the images. Moreparticularly, the processor 112 is configured to use a conventionalimage matting algorithm where the matting result is intersected with theimage regions. This allows the identification of the regions of theimage with a certain texture/edge content on which sharpness mayreliably be measured. The processor 112 employs a multi-resolutionlaplacian approach to determine the actual sharpness value. Under thisapproach, all 4 levels of the laplacian pyramid are combined in order tobe resilient to image noise. In addition, the laplacian pyramid isweighted by a non-linear function to correct the measured blur fordifferent contrast levels, since the blur perception changes with localcontrast. The correction factor based on contrast has been implementedas:

$\begin{matrix}{{SharpnessCorrectionFactor} = \{ \begin{matrix}{{{{- 0.0042} \cdot {contrast}} + 1},} & {{{for}\mspace{14mu} 0} \leq {contrast} \leq 50} & \; \\{{0.8 \cdot {\mathbb{e}}^{{- 0.024}{({{contrast} - 50})}}},} & {{{for}\mspace{14mu} 51} \leq {contrast} \leq 200.} & \;\end{matrix} } & {{Equation}\mspace{14mu}(1)}\end{matrix}$

The contrast (CN) is measured in each region using the root-mean squarecontrast. The root-mean square contrast is:

$\begin{matrix}{{{CN}_{i} = \lbrack {\frac{1}{n_{i} - 1}{\sum\limits_{\forall{j \in {region}_{1}}}( {x_{j} - \overset{\_}{x}} )^{2}}} \rbrack^{1/2}},{with}} & {{Equation}\mspace{14mu}(2)} \\{\overset{\_}{x} = {\frac{1}{n_{i}}{\sum\limits_{\forall{j \in {region}_{i}}}{x_{j}.}}}} & {{Equation}\mspace{14mu}(3)}\end{matrix}$

The colorfulness (CF) is measured with an approach which combines boththe color variance as well as the chroma magnitude in the CIE-Lab colorspace:CF _(i)=σ_(a) _(i) _(b) _(i) +0.37μ_(a) _(i) _(b) _(i) .  Equation (4)

In Equation (4), σ_(ab) is the trigonometric length of the standarddeviation in CIE-Lab space, and μ_(ab) is the distance of the center ofgravity in CIE-Lab space to the neutral color axis.

For each of the images, the processor 112 combines the sharpness,contrast, and colorfulness maps to render an appeal map (AMap) for eachof the images. In areas where there is texture/edge content, theprocessor 112 is configured to complement the sharpness metric in anadditive manner with a contribution from both the contrast and thecolorfulness. In areas that are mostly soft, for instance, contain nohigh frequencies, the contribution of contrast and colorfulness is muchlarger, that is, this will increase the image appeal measure of highlysalient regions or highly colorful regions with little texture/edgecontent. The processor 112 thus renders the appeal maps (AMap) at eachpixel (i,j) location as:

$\begin{matrix}{{{AMap}_{i,j} = {S_{i,j} + {{\alpha( S_{i,j} )} \cdot {CN}_{i,j}} + {{\beta( S_{i,j} )} \cdot {CF}_{i,j}}}},{{which}\mspace{14mu}{is}\text{:}}} & {{Equation}\mspace{14mu}(5)} \\{{\alpha( S_{i,j} )} = \{ {\begin{matrix}{\frac{1}{A + {B \cdot {SD}_{{region}\; \supset {({i,j})}}}},} & {{{for}\mspace{14mu}{SD}_{{region}\; \supset {({i,j})}}}\mspace{14mu} < {SDThres}} \\{\frac{1}{E},} & {{{{for}\mspace{14mu}{SD}_{{{region}\; \supset {({i,j})}}\mspace{14mu}}} < {SDThres}}\mspace{14mu}}\end{matrix}{and}} } & {{Equation}\mspace{14mu}(6)} \\{{\beta( S_{i,j} )} = \{ \begin{matrix}{\frac{1}{C + {D \cdot {SD}_{{region}\; \supset {({i,j})}}}},} & {{{for}\mspace{14mu}{SD}_{{region}\; \supset {({i,j})}}}\mspace{14mu} < {SDThres}} \\{\frac{1}{F},} & {{{for}\mspace{14mu}{SD}_{{{region}\; \supset {({i,j})}}\mspace{14mu}}} < {{SDThres}\;.}}\end{matrix} } & {{Equation}\mspace{14mu}(7)}\end{matrix}$

In the equations above, SD is the “sharpness density” of the specificregion to which each pixel (i,j) belongs, as the percentage of theregion being covered by the output of the matting described above. Byway of example, where SDThres=0.33, A=2, B=57, C=2, D=21, E=21, and F=9.These values substantially guarantee a higher contribution of bothcontrast and colorfulness in regions of low or no sharpness.

The processor 112 is also configured to calculate an image dependentthreshold for the final appeal metric because different images may havevery different appeal distributions. The image dependent threshold isset to one half the maximum value in the appeal map discussed abovebecause in all images, there is bound to be a more relevant area thanthe others. As such, the final appeal metric will be measured withinthis region accordingly. More particularly, the processor 112 generatesa binary map in which all of the regions with the appeal map value abovethe image dependent threshold is set to, for instance, “appealing” andall of the regions with the fuel map value below the image dependentthreshold is set to, for instance, “not appealing”.

The processor 112 may also consider an exposure measure in determiningthe final appeal metric for each of the images. Generally speaking, theprocessor 112 may impose penalties if no histogram clipping exists bothat the high and low end, and there is good coverage of most luminancevalues. According to an example, the processor 112 may employ a modelbased on the average of the luminance histogram and its standarddeviation as follows:

$\begin{matrix}{{lumFactor} = \{ {\begin{matrix}{{{if}{\mspace{14mu}\;}( {{averageLum} < {LLThres}} )},} & {A_{lum} + {( {1 - A_{lum}} ) \cdot \frac{averageLum}{LLThres}}} \\{{{if}\mspace{14mu}( {{averageLum} > {LHThres}} )},} & {1 - {B_{lum} \cdot \frac{{averageLum} - {LHThres}}{255 - {LHThres}}}} \\{{else},} & 1.\end{matrix}{And}} } & {{Equation}\mspace{14mu}(8)} \\{{sigmaFactor} = \{ \begin{matrix}{{{if}{\mspace{14mu}\;}( {{sigmaLum} < {sLLThres}} )},} & {A_{s - {lum}} + {( {1 - A_{s - {lum}}} ) \cdot \frac{sigmaLum}{sLLThres}}} \\{{{if}{\mspace{14mu}\;}( {{sigmaLum} > {sLHThres}} )},} & {1 - {B_{s - {lum}} \cdot \frac{{sigmaLum} - {sLHThres}}{255 - {sLHThres}}}} \\{{else},} & 1.\end{matrix} } & {{Equation}\mspace{14mu}(9)}\end{matrix}$

In the equations above, B_(lum)=B_(s-lum)=0.2, andA_(lum)=A_(s-lum)=0.8.

According to an example, LLThres=70, LHThres=160, sLLThres=35, andsLHThres=60.

The processor 112 may also determine the final exposure factor (E) tocomprise the product of both the luminance factor and the standarddeviation factor:E=lumFactor*sigmaFactor.  Equation (10)

The processor 112 may further consider a homogeneity measure of theappealing region in each of the images. The homogeneity measure isconsidered by thresholding the appeal map twice, once with the imagedependent threshold discussed above (½ of the maximum appeal value), andonce with one half of the first threshold (¼ of the maximum appealvalue), generating two appealing regions. With a lower threshold, theappealing region will expand to other regions of some intermediateappeal value. The more similar those two binary maps are (position andsize) the higher the homogeneity measure. That is, if the distractionsintroduced with the lower threshold are many, that is an indication thatthe appealing region is not as easy to segment out from the background,and is therefore less homogeneous.

The processor 112 may still further measure the sizes (SZ) of theappealing regions in each of the images and may consider the sizes indetermining the image appeal values of the images. For instance, theprocessor 112 may assign a greater value to those appealing regions inimages having sizes greater than a threshold. By way of example, thethreshold may be set around ⅓ of the image area. In one regard, theprocessor 112 may assign greater value to those images containing largerhomogeneous appealing regions.

The processor 112 is configured to implement an image appeal metric thatconsiders a combination of two or more of the above-described measuresin determining an image appeal level for each of the images. An imageappeal metric that results in favorable image appeal determinations isone that results in the product of the average of the appeal map overall of the appealing region times the other factors. That is, the mostappealing images are those images that have good values for all of theabove-described measures. By way of particular example, the processor112 may be configured or programmed to determine the appeal measure (AM)of each image (i) by determining the average the image appeal map (AMap)over the image appeal region only, using any combination of sharpness(S), contrast (CN), and/or colorfulness (CF/OCF), to multiply the appealaverage by the product of any combination of the other features:exposure (E), appealing region size (SZ), appealing region homogeneity(H), and colorfulness on the whole image, where N is the size of theappealing region in pixels, as noted in the following equation:

$\begin{matrix}{{AMi} = {\lbrack {\frac{1}{N}\mspace{14mu}\underset{{in}\mspace{14mu}{appealing}\mspace{14mu}{region}}{\sum\limits_{i,j}^{N}{AMap}_{i,j}}} \rbrack \cdot E \cdot {OCF} \cdot H \cdot {{SZ}.}}} & {{Equation}\mspace{14mu}(11)}\end{matrix}$

The processor 112 may thus determine the image appeal values of theimages in determining the respective relevance levels at step 204.

At step 204, the processor 112 further determines respective relevancelevels by comparing the images within each of the clusters 304, 306a-306 c to determine whether there are any images that are substantiallysimilar to each other. The processor 112 identifies the substantiallysimilar images to also determine relevance measures of the images asdiscussed below.

In determining whether images in the clusters 304, 306 a-306 c aresimilar to other images in the respective clusters 304, 306 a-306 c, theprocessor 112 may employ a similarity metric that is based on a regionbased lexical color quantization descriptor, described, for instance, inthe 61/012,099 Provisional Patent Application. As described in thatapplication, a lexical color quantization process is implemented on theimages to convert the pixel values in the images to a secondrepresentation, where the second representation has a yellow-blue axis,a red-green axis, and a luminance axis.

According to an example, the processor 112 compares the converted pixelvalues of the images with each other. More particularly, the processor112 may determine that images have a high similarity metric if the samelexical color regions in different images are similar in size andposition. It should be understood, however, that the processor 112 mayemploy other methods of determining whether images are similar withoutdeparting from a scope of the method 200.

In addition, the processor 112 may apply a similarity threshold todistinguish between when two images are sufficiently similar to warrantlabeling as being substantially similar to each other. The similaritythreshold may differ for each of the levels of the cluster hierarchy.For instance, the similarity threshold may be relaxed further closer tothe top of the hierarchy to substantially ensure larger and largersimilarity clusters closer to the top of the hierarchy.

At step 206, the processor 112 orders the images in each cluster 304,306 a-306 c according to one or more relevance measures, as shown in thehierarchical scalable image collection representation 310 in FIG. 3A. Asshown therein, the images contained in each of the clusters 304, 306a-306 c may be ranked according to a relevance representation 312. Anexample of a relevance representation 312 is depicted in greater detailin FIG. 3B.

FIG. 3B, more particularly, depicts a manner in which the processor 112classifies the images in each of the clusters 304, 306 a-306 c basedupon relevance, with one of the criteria for relevance comprising imageappeal. As shown in FIG. 3B, the relevance representation 312 includes avertically extending line labeled “image appeal 350” and a horizontallyextending line labeled “lower relevance 352”. In addition, each of thevertically extending lines represents a particular image, with theheights of the vertically extending lines representing the image appealvalue for that image. It should be understood that the relevancerepresentation 312 merely depicts one type of measure (image appeal) andthat other measures, such as, image content, image capture times, focallengths at which the images were captured, etc., may also be used todistinguish the images from each other. In this regard, the verticallyextending line 350 may comprise another type of measure withoutdeparting from a scope of the relevance representation 312.

In any regard, those images positioned closer to the left side of therelevance 352 line have the highest relevance and those imagespositioned closer to the right edge of the relevance 352 line have thelowest relevance. The relevance 352 line is also depicted as beingformed of a plurality of categories, where each of the categories has adifferent relevance value. In addition, within each of the categories,images having the highest image appeal are positioned further left alongthe lower relevance 352 line as compared with images having the lowestimage appeal. Thus, the images having the highest image appeal areconsidered to have the highest relevance in each of the categories.

The categories have been depicted as, important images 360,representative images 362, first duplicate set 364, second duplicate set366, etc. In addition, within each of the categories, the images havebeen arranged according to their respective image appeals, with thoseimages having higher image appeals being positioned further left alongthe relevance 352 line.

As shown in FIG. 3B, the images identified as being important images 360are given the highest relevance. These images may comprise, forinstance, images that the user has directly or indirectly tagged asbeing highly desirable (the most relevant images). The user may directlytag the images by indicating their high relevance through the inputsource 120. In addition, or alternatively, the images may indirectly betagged as being important based upon tracking of the user's use of theimages. By way of example, a determination of whether the user has usedthe images in other applications, for instance, emails, documents,postcards, calendars, etc., may be made and the use may be tracked.Those images that have the greatest use may automatically be tagged asbeing important images. As a further example, the desirability (orrelevance) of the images may be based upon feedback (such as tagging ofthe images) from a number of users, for instance, through a socialnetworking application or other application that enables multiple usersto access the collection of images.

The important images may also include other images that may have greaterrelevance over other images. For instance, images containing particularsmiling actors may be classified as being important images.

The representative images 362 may comprise those images that either donot have duplicates (or substantially similar images) as describedabove. The representative images 362 may also comprise images having thehighest image appeal out of duplicate images. Likewise, the images inthe first duplicate set 364 may comprise those images that have beenidentified as being duplicates (or substantially similar images), buthaving the second highest image appeal of the duplicate images.Moreover, the images in the second duplicate set 366 may comprise thoseimages that have been identified as being duplicates (or substantiallysimilar images), but having the third highest image appeal of theduplicate images. Additional duplicate images may be classified intofurther duplicate sets according to their respective image appeallevels, until all of the images in each of the clusters 304, 306 a-306 chave been assigned.

At step 208, the processor 112 arranges the images contained in theclusters 304, 306 a-306 c of one hierarchical level according to apredefined arrangement process to create a scalable relevance orderedrepresentation 118 of the image collection 116. The selection of thehierarchical level from which the images are to be arranged may beuser-selected. For instance, a user may select a particular hierarchicallevel based upon the level of detail desired from the scalablerepresentation 118. By way of example, the user may select a lower levelof the hierarchy (containing more clusters) if the user desires agreater coverage of different events, time periods, actors, etc.Alternatively, the user may select a higher level of the hierarchy(containing less clusters) if the user desires selection of a broaderrange of images.

With reference to FIG. 3A, step 208 corresponds to the creation of thescalable relevance ordered representation 320 of the images 322. Theorder in which the images 322 are selected from the clusters 304, 306a-306 c depends upon both the predefined arrangement process and theselected level of the hierarchy from which the images are selectedbecause the images may be classified into different clusters and thusdifferent categories (FIG. 3B) in the different levels of the hierarchy.For instance, an image classified as being a representative image 362 inone cluster level may be classified as being a duplicate image 364 inanother cluster level.

According to a first example, the predefined arrangement process maycomprise a user-configurable arrangement process. In this example, theuser may select both the level of the hierarchy from which the imagesare selected. In addition, the user may select one or more of theclusters to have greater prominence. The user may further remove theimages in one or more of the clusters altogether.

According to a second example, the arrangement of the images 322 in thescalable relevance ordered representation 320 is based upon a timehierarchy level selection. Under this example, the first decision is toselect the right level in a time clustering hierarchy on which tooperate, for instance, as an instruction from a user. Once the hierarchylevel has been selected, the image selection process occurs at thatlevel and the overall number of images (NN) to be selected drives theselection process. By way of example, the selection of the images (NN)may be based according to the following equations:If NN<NC ₁, then NC _(select) =NC ₁, and  Equation (12)If NC _(i-1) >NN>NC _(i), then NC _(select) =NC _(i).  Equation (13)

In the equations above, NC_(i) is the number of time clusters at eachhierarchy level i, with i being greater or equal to one. In addition,the equations above illustrate a manner in which the best time hierarchyfor the best coverage of an image collection 116 may be selected.

In instances where the user gives more prominence to a certain timecluster at the selected level, the selection for this time cluster isperformed at the immediate lower hierarchy level to enable bettercoverage of that specific event.

According to a third example, the arrangement of the images 322 in thescalable relevance ordered representation 320 is based upon a clustersize based approach. Under this approach, once the cluster hierarchylevel has been selected, images are selected from each of the clustersat that level. In addition, the selection of images alternates amongdifferent clusters based upon specific rules. For instance, thisapproach favors the clusters with more representative images in aproportional manner.

Referring to FIG. 4, there is shown a diagram 400 of a samplehierarchical scalable image collection representation 310 and scalablerelevance ordered representations 320, according to an example. Shown inthe diagram 400 are two examples of scalable relevance orderedrepresentations 320 that have been arranged using different selectionapproaches on the images contained in the clusters 306 a-306 c. Thefirst representation 320(a) depicts an arrangement based upon thecluster size based selection approach discussed above. As shown therein,under this approach, the image having the highest relevance in thelargest cluster 306 c, which is the image labeled “6”, is selectedfirst, the highest relevance image in the next largest cluster 306 b,which is the image labeled “a”, is selected next, and then the imagelabeled “c” is selected next, and so forth. This approach results in ascalable relevance ordered list of images that provides good coverage ofevents with dissimilar images, which may be a preferred manner ofstorytelling with the images.

The second representation 320(b) depicts an arrangement based upon anaverage image relevance based selection approach. Under this approach,the images having the highest average relevance in each of the clustersare selected first, regardless of cluster size. More particularly, asshown in FIG. 4, the image labeled “c” is selected first, the imagelabeled “a” is selected second, the image labeled “6” is selected third,etc. This approach favors the clusters whose average relevance of itsrepresentative images is relatively higher. In addition, this approachresults in a scalable relevance ordered list of the image collection 116with better coverage of highly relevant images, such as, images ofparticular people, particular landscapes, etc.

Generally speaking, the representation 320 is easily scalable becausethe less desirable images may easily be removed by removing the imagesfrom the right side of the representation 320. In other words, when aselected number of images is desired, the most relevant images, forinstance, as identified by a user, may be kept by simply removing imagesidentified as being less relevant. In this regard, the scalablerelevance ordered representation 320 is similar to a scalable bitstream(such as, in image compression under JPEG2000).

What has been described and illustrated herein is a preferred embodimentof the invention along with some of its variations. The terms,descriptions and figures used herein are set forth by way ofillustration only and are not meant as limitations. Those skilled in theart will recognize that many variations are possible within the scope ofthe invention, which is intended to be defined by the followingclaims—and their equivalents—in which all terms are meant in theirbroadest reasonable sense unless otherwise indicated.

What is claimed is:
 1. A method of automatically creating a scalablerelevance ordered representation of an image collection, said methodcomprising: classifying the images in the image collection into aplurality of clusters based upon a feature of the images; determining arespective relevance level of each of the images contained in each ofthe plurality of clusters, comprising: rendering a first appeal map ofeach image using a first image dependent threshold value of a metric todetermine a first appealing region; rendering a second appeal map ofeach image using a second image dependent threshold value of the metricto determine a second appealing region; determining a respectiveappealing region homogeneity measure of each image based on a similarityof the first and second appealing regions; and determining therespective relevance level of each image based at least upon therespective appealing region homogeneity measure; ordering the images ineach of the plurality of clusters according to the respective relevancelevels; and arranging the images from the ordered plurality of clustersaccording to a predefined arrangement process to create the scalablerelevance ordered representation of the image collection.
 2. The methodaccording to claim 1, wherein the feature comprises at least one of atime period, a content, a geographical position system based coordinate,and metadata of the images.
 3. The method according to claim 1, furthercomprising: receiving input from a user identifying how the relevancelevels of the images are to be determined, wherein the relevance levelsare measured based upon the input received from the user.
 4. The methodaccording to claim 1, wherein determining respective relevance levels ofthe images contained in each of the plurality of clusters furthercomprises determining at least one of smiles, faces, and image appealvalues of the images to determine the relevance levels.
 5. The methodaccording to claim 1, wherein the respective relevance level of imagescontained in each cluster of the plurality of clusters comprises ameasure of a similarity level between two or more images in eachcluster, said method further comprising: for each of the clusters,determining whether a plurality of the images are substantially similarto each other; in response to a determination that at least two of theimages being substantially similar to each other, categorizing one ofthe substantially similar images as a representative image andcategorizing another one of the substantially similar images as aduplicate image; and wherein ordering the images further comprisesordering the images such that the images categorized as representativeimages have greater relevance as compared with the images categorized asduplicate images.
 6. The method according to claim 5, furthercomprising: determining an image appeal value for each of the imagesdetermined to be substantially similar to each other; ranking the imagesdetermined to be substantially similar to each other according to theirrespective image appeal values and wherein categorizing one of thesubstantially similar images as being a representative image furthercomprises categorizing the image having the highest image appeal valueas the representative image; and wherein categorizing another one of thesubstantially similar images as a duplicate image further comprisescategorizing the images having less than the highest image appeal valuesinto one or more duplicate sets that are hierarchically arrangedaccording to their image appeal values.
 7. The method according to claim6, further comprising: ranking images determined to be important ashaving the highest relevance levels.
 8. The method according to claim 5,wherein determining whether any of the images in each of the clustersare substantially similar to each other further comprises determiningwhether any of the images in each of the clusters are substantiallysimilar to each other through a lexical quantization process of each ofthe images.
 9. The method according to claim 1, wherein determiningrespective relevance levels of the images further comprises determiningimage appeal values for each of the images based upon a combination ofsharpness, contrast, colorfulness, and at least one of exposure,appealing region size, and the appealing region homogeneity measure ofeach of the images.
 10. A computing apparatus for creating a scalablerelevance ordered representation of an image collection, said apparatuscomprising: a memory storing the image collection; a processor to:access the memory; classify the images in the image collection into aplurality of clusters based upon a feature of the images; determine arespective relevance level of each of the images contained in each ofthe plurality of clusters, wherein to determine the respective relevancelevel, the processor is to: render a first appeal map of each imageusing a first image dependent threshold value of a metric to determine afirst appealing region; render a second appeal map of each image using asecond image dependent threshold value of the metric to determine asecond appealing region; determine a respective appealing regionhomogeneity measure of each image based on a similarity of the first andsecond appealing regions; and determine the respective relevance levelof each image based at least upon the respective appealing regionhomogeneity measure; order the images in each of the plurality ofclusters according to the respective relevance levels; and arrange theimages from the ordered plurality of clusters according to a predefinedarrangement process to create the scalable relevance orderedrepresentation of the image collection from the arranged images.
 11. Thecomputing apparatus according to claim 10, wherein the processor isfurther to determine at least one of smiles, faces, and image appealvalues of the images to determine relevance levels of the images. 12.The computing apparatus according to claim 10, wherein the respectiverelevance level of images contained in each cluster of the plurality ofclusters comprises a measure of a similarity level between two or moreimages in each cluster, wherein said processor is further to, for eachof the clusters, determine whether a plurality of the images aresubstantially similar to each other, to categorize one of thesubstantially similar images as a representative image and to categorizeanother one of the substantially similar images as a duplicate image,and wherein the processor is further to order the images such that theimages categorized as representative images have greater relevance ascompared with the images categorized as duplicate images.
 13. Thecomputing apparatus according to claim 12, wherein the processor isfurther to determine an image appeal value for each of the imagesdetermined to be substantially similar to each other, to rank the imagesdetermined to be substantially similar to each other according to theirrespective image appeal values, to categorize the image having thehighest image appeal value as the representative image, and tocategorize the images having less than the highest image appeal valuesinto one or more duplicate sets that are hierarchically arrangedaccording to their image appeal values.
 14. A non-transitory computerreadable storage medium on which is embedded one or more computerprograms, said one or more computer programs implementing a method ofautomatically creating a scalable relevance ordered representation of animage collection, said one or more computer programs comprising computerreadable code to cause a computer to: classify the images in the imagecollection into a plurality of clusters based upon a feature of theimages; determine a respective relevance level of each of the imagescontained in each of the plurality of clusters, wherein to determine therespective relevance level of each image, the computer is to: render afirst appeal map of each image using a first image dependent thresholdvalue of a metric to determine a first appealing region; render a secondappeal map of each image using a second image dependent threshold valueof the metric to determine a second appealing region; determine arespective appealing region homogeneity measure of each image based on asimilarity of the first and second appealing regions; and determine therespective relevance level of each image based at least upon therespective appealing region homogeneity measure; order the images ineach of the plurality of clusters according to the respective relevancelevels; and arrange the images from the ordered plurality of clustersaccording to a predefined arrangement process to create the scalablerelevance ordered representation of the image collection.
 15. Thenon-transitory computer readable storage medium according to claim 14,said one or more computer programs further comprising computer readablecode to cause the computer to: determine at least one of smiles, faces,and image appeal values of the images to determine the relevance levels.16. A method of automatically creating a relevance orderedrepresentation of an image collection, said method comprising:determining an image appeal map for each pixel of each image in theimage collection based upon a combination of appeal metric values of atleast two image appeal metrics; determining an image appealing region ofeach image based upon an image dependent threshold value for the imageappeal map; determining an average appeal value corresponding to anaverage of the image appeal map over the image appealing region;multiplying the determined average appeal value by a combination ofvalues associated with other image appeal metrics for each image toobtain an image appeal value of each image; determining respectiverelevance levels of each image based on the image appeal value of eachimage; and arranging the images of the image collection according to thedetermined respective relevance levels of the images to create therelevance ordered representation of the image collection.
 17. The methodaccording to claim 16, wherein determining an image appeal map for eachpixel of each image comprises determining the image appeal map basedupon any combination of a sharpness metric, a contrast metric, and acolorful metric, and wherein multiplying the determined average appealvalue by a combination of values associated with other image appealmetrics includes multiplying the average appeal value by a combinationof an exposure metric, appealing region size, an appealing regionhomogeneity, and colorfulness on the entirety of each image.